Pith. sign in

REVIEW 4 major objections 5 minor 40 references

Eyes on the Image: Gaze Supervised Multimodal Learning for Chest X-ray Diagnosis and Report Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding radiologist eye-tracking as a training signal improves chest X-ray classification (F1 0.597 to 0.631, AUC 0.821 to 0.849) and produces attention maps aligned with expert gaze.

desk verdict The paper's central claim—that gaze supervision improves classification—is unsupported because the abstract's numbers come from a column that also fine-tunes on validation data; the isolated gaze loss shows no gain. read the letter →

arxiv 2508.13068 v1 pith:XVH434MR submitted 2025-08-18 cs.CV cs.LG

Tanjim Islam Riju , Shuchismita Anwar , Saman Sarker Joy , Farig Sadeque , Swakkhar Shatabda This is my paper · ORCID
classification cs.CVcs.LG
keywords chestX-raygazesupervisioneye-trackingcontrastivelearningradiologyreportgenerationmulti-labelclassificationattentionalignmentMIMIC-Eye
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that radiologist eye-tracking, used as a training-time spatial signal, makes a chest X-ray classifier both more accurate and more interpretable, and that the same predicted regions can anchor automatic report generation. Using a curated subset of the MIMIC-Eye dataset, it reports that adding gaze supervision raises macro F1 from 0.597 to 0.631 and AUC from 0.821 to 0.849, with attention maps that correlate more closely with radiologist fixations. The report-generation half converts classifier outputs into confidence-weighted keywords, maps them to 17 canonical anatomical regions, and asks an LLM to write region-aware findings and impression sections. If these results hold, gaze data would offer a practical route to explainable and clinically grounded radiology AI without dense pixel-level annotations.

What carries the argument

The load-bearing mechanism is the gaze-attention loss and the contrastive alignment that carries it. The loss combines four terms on predicted versus radiologist attention maps: MSE for pixel-level fidelity, KL divergence for distribution shape, a Pearson correlation term for pattern similarity, and a normalized center-of-mass term for geometric centering, with a per-sample quality weight that scales with fixation count and quality. An InfoNCE objective simultaneously aligns image-text and gaze-text embeddings, and a lightweight decoder predicts attention at three spatial scales. The second-stage machinery is a curated dictionary of 17 thoracic regions with alias lists and a condition-to-region clinical matrix, plus an LLM prompt that turns confidence-weighted keywords and region flags into findings and impression text.

What would settle it

Run the pipeline twice on the REFLACX half of the corpus, once with YOLO-completed bounding boxes and once with radiologist-drawn abnormality ellipses or manually corrected boxes replacing them, then compare macro F1, AUC, and attention-alignment metrics. If the synthetic boxes are noise-free, results should stay roughly equal; a systematic disagreement between the two runs, or a large measured mismatch between YOLO boxes and human annotations, would show the load-bearing assumption does not hold.

Watch

Extended reading notes

Core claim

The paper's central claim is that human gaze can serve as an effective spatial teacher for a multimodal disease classifier, not merely as an extra input feature. It proposes a two-stage framework: a gaze-guided contrastive learner that fuses image, bounding-box mask, text, and fixation embeddings with an InfoNCE objective and a novel multi-term gaze-attention loss, followed by a region-grounded report generator that maps predicted keywords to anatomical regions and prompts an LLM. It claims that supervised attention maps align with radiologist fixations (for example, Jensen-Shannon divergence 0.451 versus an inter-reader upper bound near 0.45) while classification improves, and that the generated reports reach BERTScore 0.732 and CheXpert-F1 0.546, with phrase-level factual grounding still limited at RadGraph-F1 0.129. The authors state that test-time inference uses only the image, so gaze is purely a training signal.

Load-bearing premise

The load-bearing premise is that the automatically completed location boxes for the REFLACX studies are accurate enough to be treated as expert ground truth, because both the gaze-attention training and the anatomical region mapping for reports rely on those boxes.

Editorial extensions

If this is right

  • A chest X-ray classifier can be trained to use radiologist gaze as a weak spatial label and then run image-only at test time, so gaze collection is not needed at deployment.
  • The same attention maps can be scored against radiologist fixations, giving a quantitative, human-interpretable check on where the model is looking.
  • A keyword-condition dictionary with 17 anatomical regions plus a structured LLM prompt can generate reports whose semantics match reference reports while still falling short on fine-grained factual relations.
  • Merging just 493 validation studies into training restores and improves performance, suggesting modest amounts of human annotation can sharpen a gaze-supervised model.
  • Because all training signals come from the public MIMIC-Eye data and intermediate predictions are retained, the pipeline offers a reproducible testbed for gaze-aware radiology AI.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the gaze-text InfoNCE loss suggests a way to distill eye-tracking into the text encoder, so models trained with gaze could inherit spatial priors even when deployed without an eye-tracker.
  • Beyond the paper, the claimed attention maps could serve as a filter for LLM output: only report sentences whose named anatomical region appears in the activated region mask would be kept, directly attacking the RadGraph-F1 gap the paper reports.
  • Beyond the paper, the same four-term gaze loss is modality-agnostic and could be applied to other eye-tracking corpora, such as CT or ultrasound, since none of its terms depends on chest-radiograph structure.
  • Beyond the paper, the ablation results leave open whether the classification gain comes from the spatial attention loss or from simply having more gaze-derived embeddings; a controlled study separating those two contributions would clarify the mechanism.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a two-stage multimodal framework for chest X-ray classification and report generation using the MIMIC-Eye dataset. Stage 1 fuses image, bounding-box, text, and gaze modalities through contrastive learning with a newly proposed multi-term gaze-attention loss combining MSE, KL divergence, correlation, and center-of-mass alignment. Stage 2 extracts diagnostic keywords, maps them to a 17-region anatomical dictionary, and prompts LLMs to generate region-aligned reports. The authors claim that incorporating fixations improves classification F1 from 0.597 to 0.631 and AUC from 0.821 to 0.849, and that the pipeline improves report quality as measured by clinical keyword recall and ROUGE overlap.

Significance. If the claims were supported, gaze-supervised contrastive learning with interpretable attention maps would be a useful contribution to medical multimodal learning, and a fully public-data pipeline with intermediate outputs and LLM comparisons would aid reproducibility. The manuscript has genuine strengths: it uses publicly available data, compares five LLMs, documents hyperparameters and preprocessing in appendices, acknowledges its own limitations, and frames the gaze-attention loss as a testable design. However, the headline classification improvement is confounded in Table 1, and the report-generation evaluation is circular because the prompt injects keywords extracted from the reference reports and then measures lexical overlap against those same references. These are load-bearing issues for both central claims, so the significance as presented is not established.

major comments (4)
  1. [Section 4.2, Table 1] The abstract's headline improvement (F1 0.597 to 0.631, AUC 0.821 to 0.849) pairs the '+Transcription' column with the '+Fine Tune' column. These two columns differ by two interventions: the proposed multi-term gaze-attention loss and the addition of 493 validation samples to training. The '+Fixation Enhanced' column, which isolates the proposed gaze-attention loss, reports F1 0.597 and AUC 0.823, essentially identical to the '+Transcription' baseline (F1 0.597, AUC 0.822). The claim that incorporating fixations improves classification is therefore not supported by Table 1; the gain is attributable to the fine-tuning protocol. The authors should compare models under an identical training data and protocol, or revise the abstract and conclusions to report the isolated effect of gaze supervision.
  2. [Section 4.2] The statement that Pearson r = 0.237 ± 0.191 "meets Cohen's moderate threshold (r ≥ 0.30)" is internally inconsistent: 0.237 is below 0.30, and the ±0.191 interval extends below 0.05. This overstates the human-model alignment evidence and should be corrected, along with the related claim in the Conclusion that Pearson ≈ 0.25 indicates improved alignment.
  3. [Section 3.5, Appendix A.6.3.1, Table 3] The report-generation evaluation is circular. Section 3.5 extracts diagnostic keywords from the reference reports using Gemini; the prompt in Appendix A.6.3.1 instructs the LLM to use exact phrases from CLINICAL KEYWORDS verbatim and to maximize lexical similarity to the expert reference; Table 3 then scores the generated reports against the same reference reports. Lexical-overlap metrics such as ROUGE, BERTScore, and CheXpert F1 therefore measure prompt compliance with reference-derived vocabulary rather than the model's ability to generate a faithful report from the image. The pipeline should be evaluated with keyword extraction performed on data that is not the evaluation reference, or with a prompt that does not inject reference-derived terms.
  4. [Section 4.1, Appendix A.2] The YOLO-completed bounding boxes for REFLACX are load-bearing for the report-generation stage: they supervise the gaze-attention maps and define the anatomical regions used to ground generated reports. The manuscript acknowledges that these synthetic boxes may bias spatial supervision, but no sensitivity analysis is provided. At minimum, the authors should report the quality of the YOLO completion (e.g., detection accuracy against held-out EyeGaze boxes) and assess whether the report-generation and attention-alignment results change when the completion is degraded or omitted.
minor comments (5)
  1. [Section 4.1] The partition sizes 1,984/493/400 are labeled 80.1%/19.9%/16.2%; these percentages do not sum to 100 and do not match the stated total of 2,877 aligned samples. Please clarify the denominators.
  2. [Table 1] The table caption says "Signal classification performance"; this appears to be a typo for "Multimodal classification performance" or similar.
  3. [Table 3] RadGraph-F1 values around 0.13 are reported as differences between models; without significance tests or confidence intervals on the comparison, the claim that one model "attains the highest RadGraph-F1" is not supported.
  4. [Appendix A.1] The appendix reports several different corpus sizes (3,192 patients, 3,689 studies, 2,877 aligned samples, 63 full-multimodal records); a flow diagram reconciling these counts would help reproducibility.
  5. [Table 1, p-value row] The p-value row is reported as mean ± standard deviation, which is not a meaningful summary for p-values; report the test statistic and confidence interval instead.

Circularity Check

2 steps flagged · score 5.0 of 10

Abstract's gaze gain is confounded by the validation fine-tune; report ROUGE is optimized by construction.

  1. fitted input called prediction [Abstract vs. Table 1 and Section 4.2; fine-tune definition in Section 3.4]
    "Incorporating fixations improves F1 score from 0.597 to 0.631 (+5.70%) and AUC from 0.821 to 0.849 (+3.41%). ... Fine-tuning on 493 validation samples restores peak performance (AUC = 0.849, F1=0.631). Table 1: '+ Fixation Enhanced' F1 0.597 AUC 0.823; '+ Fine Tune (Val→Train,n=493)' F1 0.631 AUC 0.849."

    The abstract computes its headline deltas by pairing the '+Transcription' baseline (F1 0.597, AUC 0.822) with the '+Fine Tune' column (F1 0.631, AUC 0.849). Those two columns differ by two interventions: the proposed gaze-attention loss and an extra fine-tuning pass on the 493-case validation split merged into training. The '+Fixation Enhanced' column, which isolates the proposed multi-term gaze loss from the fine-tune pass, reports F1 0.597 and AUC 0.823, essentially identical to the transcription baseline. Therefore the claimed improvement from 'incorporating fixations' is attributable to the validation fine-tune rather than to gaze supervision; the headline prediction reduces to a fitted input (fine-tuning on the validation subset) while being presented as a gaze effect.

  2. self definitional [Section 3.5 (keyword extraction) and Appendix A.6.3.1 (prompt template) vs. Section 4.3 (report evaluation)]
    "Gemini 2.5 Pro reads the full report and the eight target pathologies, then proposes a ranked list for each condition... The final vocabulary ... is used for anatomical-region matching and structured report generation. Prompt: 'Prefer using exact phrases from CLINICAL KEYWORDS to improve alignment with ground truth... Maximize lexical similarity to expert reference reports.' Table 3 reports scores on 400 test reports across five LLMs."

    The keyword vocabulary that conditions report generation is extracted by reading the full reference reports, and the generation prompt explicitly instructs the LLM to incorporate those exact phrases verbatim and to maximize lexical similarity with expert reference reports. Section 4.3 then scores the generated reports with ROUGE, CheXpert-F1, and related lexical-overlap metrics against reference reports. The generation objective and the evaluation metric are therefore the same construction: reference-derived terminology is injected into the generated text, and the resulting lexical overlap is reported as report quality. The report-generation 'prediction' is thus partly self-definitional rather than an independent measure of clinical utility.

full rationale

The classification stage is trained against external human fixation maps and CheXpert labels, so the core architecture itself is not circular, and no load-bearing self-citation or imported uniqueness theorem appears. However, the abstract's headline improvement is constructed by comparing the transcription baseline to the fine-tuned column, which differs by the validation fine-tune as well as the gaze loss; the '+Fixation Enhanced' ablation shows no gain over the transcription baseline (F1 0.597 vs 0.597; AUC 0.823 vs 0.822), so the '+5.70%/+3.41%' claim reduces to the fine-tune pass on 493 validation samples rather than to fixations. In the report-generation stage, the keyword vocabulary is extracted from reference reports and the prompt explicitly instructs verbatim use of those keywords to maximize lexical similarity; the subsequent ROUGE and CheXpert-F1 evaluation then measures the same lexical overlap, making the generation-quality claim partially self-definitional. Separately, Section 4.2 states Pearson r = 0.237 ± 0.191 'meets Cohen's moderate threshold (r ≥ 0.30)', which is internally inconsistent (0.237 < 0.30 and the interval extends below 0.10); this is a correctness error rather than circularity. The acknowledged YOLO bounding-box completion for REFLACX is an additional risk for region grounding, but not a circularity per se. Net: two partial construction/reduction issues justify a score of 5.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

No new physical entities are introduced. The central claim rests on hand-set hyperparameters (loss weights, fusion weight, confidence thresholds), domain assumptions about gaze and YOLO-completed boxes, and a curated anatomical knowledge matrix. The report generation stage also relies on keywords extracted from the reference corpus, which introduces a mild circularity in evaluation.

free parameters (6)
  • alpha = 0.7
    Fusion weight between global and condition-specific classification heads in Eq. 3, chosen by hand without sensitivity analysis.
  • lambda_1, lambda_2, lambda_3 = 0.1, 0.3, 0.15
    Weights for InfoNCE, gaze-attention, and gaze-text losses in Eq. 5, chosen by hand.
  • InfoNCE temperature tau = 0.07
    Temperature in the contrastive loss in Eq. 4, standard value from CLIP literature.
  • Report generation threshold = p(c) > 0.60
    Conditions above this confidence are included in the generated report (Section 3.6), hand-set.
  • Fixation quality weight w_q = sqrt(N_fix) * q_score
    Sample-adaptive weight in the gaze loss (Section 3.4), depends on fixation count and quality score, chosen by hand.
  • YOLO confidence and IoU thresholds = 0.05, 0.5
    Thresholds for bounding-box completion in Appendix A.2, chosen to maximize sensitivity and may bias spatial supervision.
assumptions (6)
  • standard math InfoNCE and focal loss are valid objectives for multimodal alignment and multi-label classification.
    Used in Eq. 4 and the classification loss in Section 3.4, standard objectives from the cited literature.
  • domain assumption Radiologist fixation maps are a reliable proxy for diagnostically relevant image regions.
    The gaze-attention loss in Eq. 5 supervises model attention with human fixation heatmaps; if fixations are not aligned with pathology, the supervision is misguided.
  • domain assumption YOLOv8n-completed bounding boxes approximate radiologist region annotations for REFLACX.
    Section 4.1 and Appendix A.2 use YOLO predictions to fill missing REFLACX region masks; these synthetic boxes are used in training and region mapping.
  • domain assumption Gemini-extracted keywords from development-set reports are condition-specific and transferable to test reports.
    Section 3.5 derives a per-condition keyword vocabulary from reports; the LLM is then prompted to use these keywords verbatim during evaluation.
  • domain assumption ROUGE, CheXpert-F1, and RadGraph-F1 are meaningful measures of clinical report quality.
    Section 4.3 uses these metrics to compare report generators without clinician evaluation or a baseline without the proposed grounding pipeline.
  • ad hoc to paper The 17-region anatomical dictionary and condition-to-region weights in Table 13 reflect clinical priors.
    The region set and weights are curated by the authors and used to map keywords to regions for report generation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Eyes on the Image: Gaze Supervised Multimodal Learning for Chest X-ray Diagnosis and Report Generation." pith.science (2026). https://pith.science/paper/XVH434MR

@misc{pith2026250813068,
  author       = {Pith},
  title        = {Pith review of: Eyes on the Image: Gaze Supervised Multimodal Learning for Chest X-ray Diagnosis and Report Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XVH434MR}},
  note         = {Machine review of arXiv:2508.13068}
}
read the original abstract

We propose a two-stage multimodal framework that enhances disease classification and region-aware radiology report generation from chest X-rays, leveraging the MIMIC-Eye dataset. In the first stage, we introduce a gaze-guided contrastive learning architecture for disease classification. It integrates visual features, clinical labels, bounding boxes, and radiologist eye-tracking signals and is equipped with a novel multi-term gaze-attention loss combining MSE, KL divergence, correlation, and center-of-mass alignment. Incorporating fixations improves F1 score from 0.597 to 0.631 (+5.70%) and AUC from 0.821 to 0.849 (+3.41%), while also improving precision and recall, highlighting the effectiveness of gaze-informed attention supervision. In the second stage, we present a modular report generation pipeline that extracts confidence-weighted diagnostic keywords, maps them to anatomical regions using a curated dictionary constructed from domain-specific priors, and generates region-aligned sentences via structured prompts. This pipeline improves report quality as measured by clinical keyword recall and ROUGE overlap. Our results demonstrate that integrating gaze data improves both classification performance and the interpretability of generated medical reports.

Figures

Figures reproduced from arXiv: 2508.13068 by the authors.

Figure 1
Figure 1. Overview of the proposed multi-modal pipeline. Images, bounding-box masks, free-text reports, and eye-tracking fixations are aligned through a contrastive objective; downstream modules generate region-grounded radiology reports. 2 Related Work Recent medical vision-language models couple large-scale chest-X-ray corpora with transformer backbones to align image embeddings and report tokens (Lu and Wang, 2025; You et … view at source ↗
Figure 2
Figure 2. Multimodal chest-X-ray classifier: image, bounding-box, text, and gaze features are independently encoded, concatenated, refined by a cross-modal attention block, and passed through an MLP fusion network whose global and condition-specific heads are ensembled to predict eight disease labels. 3.2.1 Gaze -Model Variants: Baseline vs. Enhanced We instantiate two configurations that differ solely in how they exploit fix… view at source ↗
Figure 3
Figure 3. Human -model attention alignment. Radiologist fixation map (left) vs. Fixation-Enhanced saliency (right) on the same case; matching hotspots (lower right lung, paracardiac) show gaze-guided supervision targets clinically relevant regions. Per-Condition Performance [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Pearson correlation coefficients between binary condition labels prior to pruning. Strong negative associations [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]
Figure 5
Figure 5. Figure 5: t-SNE class-specific overlays. Each subplot highlights the distribution of a given condition: (A.3) Atelectasis, [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 18 canonical work pages

  1. [1]

    A systematic review of natural language processing applied to radiology reports

    Arlene Casey, Emma Davidson, Michael Poon, Hang Dong, Daniel Duma, Andreas Grivas, Claire Grover, V \'i ctor Su \'a rez-Paniagua, Richard Tobin, William Whiteley, Honghan Wu, and Beatrice Alex. A systematic review of natural language processing applied to radiology reports. BMC Medical Informatics and Decision Making, 21 0 (1): 0 179, 2021. ISSN 1472-6947...

    work page doi:10.1186/s12911-021-01533-7 2021

  2. [2]

    Clinically accurate chest x-ray report generation, 2019

    Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew McDermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi. Clinically accurate chest x-ray report generation, 2019. URL https://arxiv.org/abs/1904.02633

    arXiv 2019

  3. [3]

    Interactive and explainable region-guided radiology report generation

    Tim Tanida, Philip M\"uller, Georgios Kaissis, and Daniel Rueckert. Interactive and explainable region-guided radiology report generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7433--7442, June 2023

    work page 2023

  4. [4]

    Kevin Zhou, and Li Xiao

    Shuxin Yang, Xian Wu, Shen Ge, Zhuozhao Zheng, S. Kevin Zhou, and Li Xiao. Radiology report generation with a learned knowledge base and multi-modal alignment. Medical Image Analysis, 86: 0 102798, 2023. ISSN 1361-8415. doi:https://doi.org/10.1016/j.media.2023.102798. URL https://www.sciencedirect.com/science/article/pii/S1361841523000592

    arXiv 2023

  5. [5]

    Wu, Arjun Sharma, Matthew Tong, Shafiq Abedin, David Beymer, Vandana Mukherjee, Elizabeth A

    Alexandros Karargyris, Satyananda Kashyap, Ismini Lourentzou, Joy T. Wu, Arjun Sharma, Matthew Tong, Shafiq Abedin, David Beymer, Vandana Mukherjee, Elizabeth A. Krupinski, and Mehdi Moradi. Creation and validation of a chest x-ray dataset with eye-tracking and report dictation for ai development. Scientific Data, 8 0 (1): 0 92, 2021. ISSN 2052-4463. doi:...

    doi:10.1038/s41597-021-00863-5 2021

  6. [6]

    Auffermann, Jessica Chan, Phuong-Anh T

    Ricardo Bigolin Lanfredi, Mingyuan Zhang, William F. Auffermann, Jessica Chan, Phuong-Anh T. Duong, Vivek Srikumar, Trafton Drew, Joyce D. Schroeder, and Tolga Tasdizen. REFLACX, a dataset of reports and eye-tracking data for localization of abnormalities in chest x-rays . Scientific Data, 9 0 (1): 0 350, 2022. ISSN 2052-4463. doi:10.1038/s41597-022-01441...

    doi:10.1038/s41597-022-01441-z 2022

  7. [7]

    Eye-gaze guided multi-modal alignment for medical representation learning

    Chong Ma, Hanqi Jiang, Wenting Chen, Yiwei Li, Zihao Wu, Xiaowei Yu, Zhengliang Liu, Lei Guo, Dajiang Zhu, Tuo Zhang, Dinggang Shen, Tianming Liu, and Xiang Li. Eye-gaze guided multi-modal alignment for medical representation learning. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Informa...

    work page 2024

  8. [8]

    Integrating language into medical visual recognition and reasoning: A survey

    Yinbin Lu and Alan Wang. Integrating language into medical visual recognition and reasoning: A survey. Medical Image Analysis, 102: 0 103514, 2025. ISSN 1361-8415. doi:https://doi.org/10.1016/j.media.2025.103514. URL https://www.sciencedirect.com/science/article/pii/S1361841525000623

    arXiv 2025

Show all 40 references
  1. [9]

    Hong, Woonhyuk Baek, and Byungseok Roh

    Kihyun You, Jawook Gu, Jiyeon Ham, Beomhee Park, Jiho Kim, Eun K. Hong, Woonhyuk Baek, and Byungseok Roh. Cxr-clip: Toward large scale chest x-ray language-image pre-training. In Hayit Greenspan, Anant Madabhushi, Parvin Mousavi, Septimiu Salcudean, James Duncan, Tanveer Syeda...

    2023 doi

  2. [10]

    Medclip: Contrastive learning from unpaired medical images and text

    Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3876--3887. Association for Computational Linguist...

    2022 doi

  3. [11]

    Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P

    Shruthi Bannur, Stephanie Hyland, Qianchu Liu, Fernando P\'erez-Garc \' a, Maximilian Ilse, Daniel C. Castro, Benedikt Boecking, Harshita Sharma, Kenza Bouzid, Anja Thieme, Anton Schwaighofer, Maria Wetscherek, Matthew P. Lungren, Aditya Nori, Javier Alvarez-Valle, and Ozan Ok...

    2023

  4. [12]

    Knowledge-enhanced visual-language pre-training on chest radiology images

    Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Weidi Xie, and Yanfeng Wang. Knowledge-enhanced visual-language pre-training on chest radiology images. Nature Communications, 14 0 (1): 0 4542, 2023. ISSN 2041-1723. doi:10.1038/s41467-023-40260-7. URL https://doi.org/10.1038/s41467-023-40260-7

    2023 doi

  5. [13]

    Enhancing medical vision-language contrastive learning via inter-matching relation modeling

    Mingjian Li, Mingyuan Meng, Michael Fulham, David Dagan Feng, Lei Bi, and Jinman Kim. Enhancing medical vision-language contrastive learning via inter-matching relation modeling. IEEE Transactions on Medical Imaging, 44 0 (6): 0 2463--2476, 2025. doi:10.1109/TMI.2025.3534436

    2025

  6. [14]

    Gazegnn: A gaze-guided graph neural network for chest x-ray classification

    Bin Wang, Hongyi Pan, Armstrong Aboah, Zheyuan Zhang, Elif Keles, Drew Torigian, Baris Turkbey, Elizabeth Krupinski, Jayaram Udupa, and Ulas Bagci. Gazegnn: A gaze-guided graph neural network for chest x-ray classification. In Proceedings of the IEEE/CVF Winter Conference on A...

    2024

  7. [15]

    Act like a radiologist: Radiology report generation across anatomical regions

    Qi Chen, Yutong Xie, Biao Wu, Xiaomin Chen, James Ang, Minh-Son To, Xiaojun Chang, and Qi Wu. Act like a radiologist: Radiology report generation across anatomical regions. In Proceedings of the Asian Conference on Computer Vision (ACCV), pages 1--17, December 2024

    2024

  8. [16]

    Look & mark: Leveraging radiologist eye fixations and bounding boxes in multimodal large language models for chest X -ray report generation

    Yunsoo Kim, Jinge Wu, Su Hwan Kim, Pardeep Vasudev, Jiashu Shen, and Honghan Wu. Look & mark: Leveraging radiologist eye fixations and bounding boxes in multimodal large language models for chest X -ray report generation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and ...

    2025

  9. [17]

    Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez Valle, Hoifung Poon, and Ozan Oktay

    Benedikt Boecking, Naoto Usuyama, Sruthi Bannur, Daniel Coelho de Castro, Anton Schwaighofer, Stephanie Hyland, Himanshu Sharma, Moritz T. Wetscherek, Tristan Naumann, Aditya Nori, Javier Alvarez Valle, Hoifung Poon, and Ozan Oktay. Ms-cxr: Making the most of text semantics to...

    2024 doi

  10. [18]

    Contrastive multimodal fusion with tupleinfonce

    Yunze Liu, Qingnan Fan, Shanghang Zhang, Hao Dong, Thomas Funkhouser, and Li Yi. Contrastive multimodal fusion with tupleinfonce. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 754--763, October 2021

    2021

  11. [19]

    Geras, and Farah E

    Nasir Hayat, Krzysztof J. Geras, and Farah E. Shamout. Medfuse: Multi-modal fusion with clinical time-series data and chest x-ray images. In Zachary Lipton, Rajesh Ranganath, Mark Sendak, Michael Sjoding, and Serena Yeung, editors, Proceedings of the 7th Machine Learning for H...

    2022

  12. [20]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2010 arXiv

  13. [21]

    Mong, Safwan S

    Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, Jayne Seekins, David A. Mong, Safwan S. Halabi, Jesse K. Sandberg, Ricky Jones, David B. Larson, Curtis P. Langlotz, Bhavik ...

    2019

  14. [22]

    Publicly available clinical BERT embeddings

    Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. Publicly available clinical BERT embeddings. In Anna Rumshisky, Kirk Roberts, Steven Bethard, and Tristan Naumann, editors, Proceedings of the 2nd Clinical Natural Lang...

    2019 doi

  15. [23]

    Representation learning with contrastive predictive coding, 2019

    Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019. URL https://arxiv.org/abs/1807.03748

    2019 arXiv

  16. [24]

    Gutmann and Aapo Hyv\" a rinen

    Michael U. Gutmann and Aapo Hyv\" a rinen. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. J. Mach. Learn. Res., 13 0 (1): 0 307–361, February 2012. ISSN 1532-4435

    2012

  17. [25]

    Statistical Power Analysis for the Behavioral Sciences

    Jacob Cohen. Statistical Power Analysis for the Behavioral Sciences. Routledge, 2nd edition, 1988

    1988

  18. [26]

    What do different evaluation metrics tell us about saliency models? IEEE Transactions on Pattern Analysis and Machine Intelligence, 41 0 (3): 0 740--757, 2019

    Zoya Bylinskii, Tilke Judd, Aude Oliva, Antonio Torralba, and Frédo Durand. What do different evaluation metrics tell us about saliency models? IEEE Transactions on Pattern Analysis and Machine Intelligence, 41 0 (3): 0 740--757, 2019

    2019

  19. [27]

    Peters and L

    R.J. Peters and L. Itti. Components of bottom-up gaze allocation in natural images. In IEEE CVPR, pages 1--7, 2005

    2005

  20. [28]

    Evaluating human gaze and model attention on glaucoma oct interpretation

    Wei Zhang and et al. Evaluating human gaze and model attention on glaucoma oct interpretation. Medical Image Analysis, 2024. In Press

    2024

  21. [29]

    Nascimento, Joao Pereira, Joaquim Jorge, and Catarina Moreira

    Chihcheng Hsieh, Chun Ouyang, Jacinto C. Nascimento, Joao Pereira, Joaquim Jorge, and Catarina Moreira. MIMIC-Eye: Integrating MIMIC Datasets with REFLACX and Eye Gaze for Multimodal Deep Learning Applications . https://doi.org/10.13026/pc72-as03, 2023. PhysioNet. RRID:SCR\_007345

    doi

  22. [30]

    Saliency and human fixations: State-of-the-art and study of comparison metrics

    Nicolas Riche, Matei Mancas, Thierry Dutoit, and Bernard Gosselin. Saliency and human fixations: State-of-the-art and study of comparison metrics. In IEEE ICCV Workshops, pages 1153--1160, 2013

    2013

  23. [31]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311–318, USA, 2002. Association for Computational ...

    2002

  24. [32]

    ROUGE : A package for automatic evaluation of summaries

    Chin-Yew Lin. ROUGE : A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74--81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013/

    2004

  25. [33]

    METEOR : An automatic metric for MT evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. METEOR : An automatic metric for MT evaluation with improved correlation with human judgments. In Jade Goldstein, Alon Lavie, Chin-Yew Lin, and Clare Voss, editors, Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measure...

    2005

  26. [34]

    Weinberger, and Yoav Artzi

    Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SkeHuCVFDr

    2020

  27. [35]

    Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration

    Daniel Deutsch, George Foster, and Markus Freitag. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Process...

    2023 doi

  28. [36]

    Lungren, Andrew Y

    Saahil Jain, Ashwin Agrawal, Adriel Saporta, Steven QH Truong, Du Nguyen Duong, Tan Bui, Pierre Chambon, Yuhao Zhang, Matthew P. Lungren, Andrew Y. Ng, Curtis P. Langlotz, and Pranav Rajpurkar. Radgraph: Extracting clinical entities and relations from radiology reports, 2021. ...

    2021 arXiv

  29. [37]

    O'Connor, and Xavier Gir \' o - i - Nieto

    Junting Pan, Kevin McGuinness, Elisa Sayrol, Noel E. O'Connor, and Xavier Gir \' o - i - Nieto. Shallow and deep convolutional networks for saliency prediction. CoRR, abs/1603.00845, 2016. URL http://arxiv.org/abs/1603.00845

    2016 arXiv

  30. [38]

    R. A. Fisher. Statistical Methods for Research Workers, pages 66--70. Springer New York, New York, NY, 1992. ISBN 978-1-4612-4380-9. doi:10.1007/978-1-4612-4380-9_6. URL https://doi.org/10.1007/978-1-4612-4380-9_6

    1992 doi

  31. [39]

    Menéndez, J.A

    M.L. Menéndez, J.A. Pardo, L. Pardo, and M.C. Pardo. The jensen-shannon divergence. Journal of the Franklin Institute, 334 0 (2): 0 307--318, 1997. ISSN 0016-0032. doi:https://doi.org/10.1016/S0016-0032(96)00063-4. URL https://www.sciencedirect.com/science/article/pii/S0016003...

    1997 doi

  32. [40]

    Methods for comparing scanpaths and saliency maps: strengths and weaknesses

    Olivier Le Meur and Thierry Baccino. Methods for comparing scanpaths and saliency maps: strengths and weaknesses. Behavior Research Methods, 45 0 (1): 0 251--266, 2013. ISSN 1554-3528. doi:10.3758/s13428-012-0226-9. URL https://doi.org/10.3758/s13428-012-0226-9

    2013 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.